Fix for No VMs start after Renew Host Security Keys due to wrong qemu group reading #11632

StepBee · 2025-09-13T17:21:34Z

Description

This PR fixes reading the libvirt/qemu group name from the libvirtd qemu config file including spaces and quotes,
which is used during the deployment of Host Security Keys.
This PR fixes #11144

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested renewal of Host Security Keys on Ubuntu 24.04 hosts and stopping and starting VMs afterwards.

How did you try to break this feature and the system with this change?

boring-cyborg · 2025-09-13T17:21:37Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: [email protected] (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

codecov · 2025-09-13T18:08:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.17%. Comparing base (6d16ac2) to head (b50a959).
⚠️ Report is 8 commits behind head on 4.20.

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #11632   +/-   ##
=========================================
  Coverage     16.17%   16.17%           
- Complexity    13296    13299    +3     
=========================================
  Files          5656     5656           
  Lines        498144   498151    +7     
  Branches      60437    60441    +4     
=========================================
+ Hits          80584    80587    +3     
- Misses       408590   408592    +2     
- Partials       8970     8972    +2

Flag	Coverage Δ
uitests	`4.00% <ø> (ø)`
unittests	`17.03% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

weizhouapache · 2025-09-13T18:29:08Z

@StepBee
cool, can you share the cases you have tested ?

weizhouapache · 2025-09-13T18:29:15Z

@blueorangutan package

blueorangutan · 2025-09-13T18:30:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

StepBee · 2025-09-13T18:33:30Z

@StepBee cool, can you share the cases you have tested ?

Sure @weizhouapache , i tested the following cases

group=kvm
group = kvm
group="kvm"
group = "kvm"

blueorangutan · 2025-09-13T19:44:42Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 14991

weizhouapache · 2025-09-14T06:10:52Z

@blueorangutan test

blueorangutan · 2025-09-14T06:12:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2025-09-14T06:46:14Z

@blueorangutan test ubuntu24 kvm-ubuntu24

blueorangutan · 2025-09-14T06:48:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu24 mgmt + kvm-ubuntu24) has been kicked to run smoke tests

blueorangutan · 2025-09-14T22:16:59Z

[SF] Trillian test result (tid-14327)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 54958 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11632-t14327-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2025-09-14T22:49:51Z

[SF] Trillian test result (tid-14328)
Environment: kvm-ubuntu24 (x2), zone: Advanced Networking with Mgmt server u24
Total time taken: 55371 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11632-t14328-kvm-ubuntu24.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache

lgtm

verified OK with different settings

Copilot

Pull Request Overview

This PR fixes a bug in the Host Security Keys renewal process where QEMU group name parsing from the libvirtd configuration file was failing due to improper handling of spaces and quotes. The fix ensures VMs can start properly after Host Security Keys renewal.

Updated QEMU group name extraction logic to handle spaces and quotes in configuration values
Fixed regex pattern to match group configuration lines with optional whitespace
Improved AWK parsing to correctly extract the group name value

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

scripts/util/keystore-cert-import

vishesh92

lgtm. tested different variations in the conf and qemu group is getting extracted correctly after the changes.

weizhouapache · 2025-09-15T06:57:49Z

@StepBee
the commands worked in our testing

however, @Copilot suggested to use the following , which looks more easy

QEMU_GROUP=$(sed -n 's/^group\s*=//p' /etc/libvirt/qemu.conf | tr -d '"' | tr -d ' ' | tail -n1)

also, do we consider single quote in the value, for example group = 'root' ?

vishesh92 · 2025-09-15T07:01:18Z

@StepBee the commands worked in our testing

however, @Copilot suggested to use the following , which looks more easy
QEMU_GROUP=$(sed -n 's/^group\s*=//p' /etc/libvirt/qemu.conf | tr -d '"' | tr -d ' ' | tail -n1)
also, do we consider single quote in the value, for example group = 'root' ?

I agree with @weizhouapache. IMO we should consider single quotes as well in the configuration.

scripts/util/keystore-cert-import

Simplify and correct qemu group extraction from configfile Co-authored-by: Wei Zhou <[email protected]>

StepBee · 2025-09-15T09:07:22Z

@StepBee the commands worked in our testing
however, @Copilot suggested to use the following , which looks more easy
QEMU_GROUP=$(sed -n 's/^group\s*=//p' /etc/libvirt/qemu.conf | tr -d '"' | tr -d ' ' | tail -n1)
also, do we consider single quote in the value, for example group = 'root' ?
I agree with @weizhouapache. IMO we should consider single quotes as well in the configuration.

@StepBee the commands worked in our testing
however, @Copilot suggested to use the following , which looks more easy
QEMU_GROUP=$(sed -n 's/^group\s*=//p' /etc/libvirt/qemu.conf | tr -d '"' | tr -d ' ' | tail -n1)

also, do we consider single quote in the value, for example group = 'root' ?

I agree with @weizhouapache. IMO we should consider single quotes as well in the configuration.

Thank you @vishesh92 @weizhouapache i agree, the awk was not required and it's good to catch single quote as well during this fix.

weizhouapache · 2025-09-15T09:19:09Z

@blueorangutan package

blueorangutan · 2025-09-15T09:20:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache

retest the commands with more values, looks good

thanks @StepBee for the update

blueorangutan · 2025-09-15T11:07:18Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15004

weizhouapache · 2025-09-15T11:11:02Z

@blueorangutan test

blueorangutan · 2025-09-15T11:12:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2025-09-15T11:13:53Z

@blueorangutan test ubuntu24 kvm-ubuntu24

blueorangutan · 2025-09-15T11:16:05Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu24 mgmt + kvm-ubuntu24) has been kicked to run smoke tests

blueorangutan · 2025-09-15T11:17:51Z

[SF] Trillian Build Failed (tid-14342)

weizhouapache · 2025-09-15T11:25:19Z

@blueorangutan test

blueorangutan · 2025-09-15T11:26:06Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2025-09-15T12:23:03Z

Merging based on approvals and manual tests

boring-cyborg · 2025-09-15T12:23:08Z

Awesome work, congrats on your first merged pull request!

blueorangutan · 2025-09-16T04:03:23Z

[SF] Trillian test result (tid-14343)
Environment: kvm-ubuntu24 (x2), zone: Advanced Networking with Mgmt server u24
Total time taken: 57682 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11632-t14343-kvm-ubuntu24.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	423.87	test_vpc_redundant.py

blueorangutan · 2025-09-16T05:31:50Z

[SF] Trillian test result (tid-14344)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 61609 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11632-t14344-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_redundant_VPC_default_routes	`Failure`	446.18	test_vpc_redundant.py

fix getting qemu group from config

60e64d2

weizhouapache added this to the 4.20.2 milestone Sep 13, 2025

weizhouapache self-assigned this Sep 13, 2025

weizhouapache approved these changes Sep 15, 2025

View reviewed changes

vishesh92 requested a review from Copilot September 15, 2025 06:25

Copilot AI reviewed Sep 15, 2025

View reviewed changes

scripts/util/keystore-cert-import Outdated Show resolved Hide resolved

vishesh92 approved these changes Sep 15, 2025

View reviewed changes

weizhouapache reviewed Sep 15, 2025

View reviewed changes

scripts/util/keystore-cert-import Outdated Show resolved Hide resolved

Update scripts/util/keystore-cert-import

b50a959

Simplify and correct qemu group extraction from configfile Co-authored-by: Wei Zhou <[email protected]>

weizhouapache approved these changes Sep 15, 2025

View reviewed changes

weizhouapache merged commit 0cbebbd into apache:4.20 Sep 15, 2025
26 checks passed

weizhouapache mentioned this pull request Sep 15, 2025

No VMs start after Renew Host Security Keys on newer Ubuntu and newer libvirt #11144

Closed

weizhouapache linked an issue Sep 15, 2025 that may be closed by this pull request

No VMs start after Renew Host Security Keys on newer Ubuntu and newer libvirt #11144

Closed

Fix for No VMs start after Renew Host Security Keys due to wrong qemu group reading #11632

Fix for No VMs start after Renew Host Security Keys due to wrong qemu group reading #11632

Uh oh!

Conversation

StepBee commented Sep 13, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

boring-cyborg bot commented Sep 13, 2025

Uh oh!

codecov bot commented Sep 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Sep 13, 2025

Uh oh!

weizhouapache commented Sep 13, 2025

Uh oh!

blueorangutan commented Sep 13, 2025

Uh oh!

StepBee commented Sep 13, 2025

Uh oh!

blueorangutan commented Sep 13, 2025

Uh oh!

weizhouapache commented Sep 14, 2025

Uh oh!

blueorangutan commented Sep 14, 2025

Uh oh!

weizhouapache commented Sep 14, 2025

Uh oh!

blueorangutan commented Sep 14, 2025

Uh oh!

blueorangutan commented Sep 14, 2025

Uh oh!

blueorangutan commented Sep 14, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Sep 15, 2025

Uh oh!

vishesh92 commented Sep 15, 2025

Uh oh!

Uh oh!

StepBee commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache commented Sep 15, 2025

Uh oh!

blueorangutan commented Sep 15, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Sep 15, 2025

Uh oh!

weizhouapache commented Sep 15, 2025

Uh oh!

blueorangutan commented Sep 15, 2025

Uh oh!

weizhouapache commented Sep 15, 2025

Uh oh!

blueorangutan commented Sep 15, 2025

Uh oh!

blueorangutan commented Sep 15, 2025

Uh oh!

codecov bot commented Sep 13, 2025 •

edited

Loading

StepBee commented Sep 15, 2025 •

edited

Loading